Goto

Collaborating Authors

 deepmind train agent


DeepMind Trains Agents to Control Computers as Humans Do to Solve Everyday Tasks

#artificialintelligence

While the design and development of contemporary AI systems has been largely results-oriented, there are also scenarios where it could be advantageous if models learned to do things "as a human would" to help with everyday tasks. That's the premise of the new DeepMind paper A Data-driven Approach for Learning To Control Computers, which proposes agents that can operate our digital devices via keyboard and mouse with goals specified in natural language. The study builds on recent developments in natural language processing, code production, and multimodal interactive behaviour in 3D simulated worlds that have enabled the generation of models with remarkable domain knowledge and desirable human-agent interaction capabilities. The proposed agents are trained on keyboard and mouse computer control for specific tasks with pixel and Document Object Model (DOM) observations, and achieve state-of-the-art and human-level mean performance across all tasks on the MiniWob benchmark. MiniWob is a challenging suite of web-browser-based tasks for computer control, ranging from simple button clicking to complex formfilling.